The following description includes information that may be useful in understanding the present disclosure. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.
A file server is a computer that provides file service relating to the organization of information on storage devices, such as disks. The file server or filer includes a storage operating system that implements a file system to logically organize the information as a hierarchical structure of directories and files on the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
A filer may be further configured to operate according to a client/server model of information delivery to thereby allow many clients to access files stored on a server, e.g., the filer. In this model, the client may comprise an application, such as a database application, executing on a computer that “connects” to the filer over a direct connection or computer network, such as a point-to-point link, shared local area network (LAN), wide area network (WAN), or virtual private network (VPN) implemented over a public network such as the Internet. Each client may request the services of the file system on the filer by issuing file system protocol messages (in the form of packets) to the filer over the network.
A common type of file system is a “write in-place” file system, an example of which is the conventional Berkeley fast file system. By “file system” it is meant generally a structuring of data and metadata on a storage device, such as disks, which permits reading/writing of data on those disks. In a write in-place file system, the locations of the data structures, such as inodes and data blocks, on disk are typically fixed. An inode is a data structure used to store information, such as metadata, about a file, whereas the data blocks are structures used to store the actual data for the file. The information contained in an inode may include, e.g., ownership of the file, access permission for the file, size of the file, file type and references to locations on disk of the data blocks for the file. The references to the locations of the file data are provided by pointers in the inode, which may further reference indirect blocks that, in turn, reference the data blocks, depending upon the quantity of data in the file. Changes to the inodes and data blocks are made “in-place” in accordance with the write in-place file system. If an update to a file extends the quantity of data for the file, an additional data block is allocated and the appropriate inode is updated to reference that data block.
Another type of file system is a write-anywhere file system that does not overwrite data on disks. If a data block on disk is retrieved (read) from disk into memory and “dirtied” with new data, the data block is stored (written) to a new location on disk to thereby optimize write performance. A write-anywhere file system may initially assume an optimal layout such that the data is substantially contiguously arranged on disks. The optimal disk layout results in efficient access operations, particularly for sequential read operations, directed to the disks.
As used herein, the term “storage operating system” generally refers to the computer-executable code operable on a computer that manages data access and may, in the case of a filer, implement file system semantics, such as a Write Anywhere File Layout (WAFL™) file system. The storage operating system can also be implemented as an application program operating over a general-purpose operating system, such as UNIX® or Windows NT®, or as a general-purpose operating system with configurable functionality, which is configured for storage applications as described herein.
Disk storage is typically implemented as one or more storage “volumes” that comprise physical storage disks, defining an overall logical arrangement of storage space. Currently available filer implementations can serve a large number of discrete volumes (150 or more, for example). Each volume is associated with its own file system and, for purposes hereof, volume and file system shall generally be used synonymously. The disks within a volume are typically organized as one or more groups of Redundant Array of Independent (or Inexpensive) Disks (RAID). RAID implementations enhance the reliability/integrity of data storage through the redundant writing of data “stripes” across a given number of physical disks in the RAID group, and the appropriate caching of parity information with respect to the striped data. In the example of a WAFL file system, a RAID 4 implementation is advantageously employed. This implementation specifically entails the striping of data across a group of disks, and separate parity caching within a selected disk of the RAID group. As described herein, a volume typically comprises at least one data disk and one associated parity disk (or possibly data/parity partitions in a single disk) arranged according to a RAID 4, or equivalent high-reliability, implementation.
Data storage is an increasingly crucial and central part of many industries dealing in financial transactions and other sensitive tasks, such as banks, government facilities/contractors, defense, health care institutions, pharmaceutical companies and securities brokerages. In many of these environments, it is necessary to store selected data in an immutable and unalterable manner. This need continues to grow in the light of current concerns over institutional fraud and mismanagement, wherein the temptation on the part of wrongdoers to erase or alter incriminating data is always present. Forms of data that require immutable treatment often include e-mails, financial documents and transaction records, and any other record that may act as proof of an important action or decision. Even in less-critical/unregulated environments, the ability to store a secure unalterable data cache is highly desirable. For example engineering, medical, law and other professional firms may wish to establish a cache of key data (e.g. invention reports or design files, client communications, medical images, etc.), that will remain unaltered and online for long periods on time. These caches can provide reliable references and proofs for clients and other interested parties.
For an example of a highly regulated environment, the United States Securities and Exchange Commission (SEC)—the body that regulates all securities transactions and reporting relative to public corporations—promulgates SEC Rule17 a-4 governing document retention for brokers and investment institutions. This rule requires that these entities store e-mails and other documents in connection with a variety of transactions and trades by clients of the entities unchanged and unchangeable for a number of years and to be able to provide these records to the SEC and other regulators on short notice. Failure to comply with these rules can lead to significant sanctions.
A variety of prior art approaches involving tape drives, electro-optical recordable media and the like have been employed over the years to implement a WORM storage system. Each of these systems has certain drawbacks in terms of storage size, speed maintenance requirements or a combination of these (and other) factors.
In the above-incorporated-by-reference U.S. patent application Ser. No. 10/391,245 entitled WRITE-ONCE-READ-MANY STORAGE SYSTEM AND METHOD FOR IMPLEMENTING THE SAME, a particularly advantageous approach to WORM storage is taught, which employs conventional fault-tolerant (e.g. RAID-based) disk storage (or similar rewritable media) as a platform for a WORM storage system. This described system is advantageous in that such disks are large in storage capacity, relatively inexpensive and easily added to an existing storage implementation. However, these disks are also inherently rewritable and/or erasable, in light of existing operating systems and protocols that are typically designed with semantics that specifically enable the free rewriting and erasure of attached disks. The described WORM storage approach is, therefore, specially configured to absolutely prevent alteration of any WORM-designated data. Also, to maintain longevity of the solution and make it available to as many clients as possible, the described WORM implementation utilizes open protocols such as CIFS and NFS and requires minimal alteration to these protocols or the applications that employ them and a minimal footprint on client applications. The system is, thus, organized around WORM storage volumes that contain files, which when committed to WORM storage, cannot be deleted or modified. Any file path or directory tree structure used to identify the file within the WORM volume is locked and cannot be deleted.
In the described WORM system, an administrator creates a WORM volume (or other WORM-designated data organizational structure), capable of storing designated WORM files (or other “data sets”). The client then creates an appropriate WORM file using the appropriate protocol semantics. The file is written to the volume and committed to WORM state by transitioning the file attributes from a not-read-only state to a read-only state. The file system persistently stores the WORM state of a file with the attributes and metadata for the file and uses this persistent WORM state to recognize WORM files on a WORM volume. Henceforth, any attempt to modify the file attributes, write to the file, or delete the file, by clients, administrators or other entities is rejected and a request denied message is returned to the attempting party. Since the file cannot be deleted, conventional file system semantics prevent deletion of the directory path. Likewise, the file system does not permit renaming of directories in an illustrative embodiment to thereby ensure the reliable and immutable identification of WORM files within the directory structure.
Committing of the WORM file to the WORM storage volume can be performed by the client via a command line interface in an interactive manner. Alternatively, applications, which are familiar with the WORM semantics, can be adapted to commit the file using an appropriate application program interface or other programmatic command structure. Similarly, open protocols, such as NFS or CIFS, through which the clients communicate with the file server/file system can be modified to enable automatic commit of created files upon a key event, such as closing of the file. The protocols and file system can be adapted to enable specialized WORM directories within the volume. An appropriate WORM file extension can be provided so that worm files within the volume can be readily identified by the client. Also, selected mirroring and backup functions may be allowed, while other backup functions that enable restoration or reversion of the volume to an earlier point in time may be disabled.
Many regulatory schemes governing WORM data storage (for example SEC 240.17a-4) specify provisions for retention periods, after which the WORM data can be discarded. In the absence of a specified retention period, applied to the record on creation, the regulations generally specify permanent retention. In the case of removable media, such as tapes or electro-optical storage, the media are carefully indexed and stored (often in secure sites) during their retention periods. Upon expiration of an applicable retention date, the expired media is retrieved from storage and physically destroyed. Since disk storage has the inherent ability to be rewritten and reused when a particular record is no longer needed, it is contemplated that the WORM protection on various on-disk records may carry a retention date, and when the retention date passes, the expired WORM record and associated data may be erased, thus preserving storage resources and ensuring the orderly and predictable removal of expired WORM data-without the material waste evoked by physical media destruction.
One commercially available WORM storage system marketed under the trade name Centera from EMC Corp. of Hopkinton, Mass. enables basic forms of retention dates for record storage. The system utilizes a network-connected cluster of general-purpose computer systems running a customized variant of the Linux operating system. A proprietary application programming interface (API) and proprietary protocols for interfacing with the storage system, as opposed to the open protocol and open standardized API approach is implemented by these computers. As such, applications can only access the storage and manipulate records through proprietary mechanisms or through a “gateway” interposed between the users and the storage system, which translates an open protocol to the proprietary protocols supported by the storage system.
This form of WORM storage system utilizes, so-called “Content Addressable Storage,” (CAS) for management of stored records. CAS relies on computing digital signatures, using an algorithm such as an MD5 hash, of the contents of any WORM-stored records to create a unique key (of “content address”) for each and every record. A representation of the digital signature of a record is used as the “key,” or “content address,” with which any future reference to the stored object must be made. This is often described as similar to a “claim check” system whereby the storage system generates a unique key for every object stored, which it returns to the application. The application is responsible for management and preservation of these content addresses, which must be performed external to the storage system.
To associate retention information with a stored record, the proprietary API permits metadata, in a proprietary format, to be associated with a stored object. This metadata information can include retention information for the record. The API supports the ability to extend retention dates further into the future, and in certain configurations, to assign an infinite retention date to those records submitted without retention information. Because of the CAS architecture, every object written to the system, as long as it has unique contents, is stored as a unique object with a unique content address. To enable WORM functionality the API prevents deletion of objects prior to the expiration of their associated retention period. Modification of existing objects is impossible because any changes in the contents of an object will result in a new content address, and hence a new object being created in the storage system.
To track retention time and other time-dependent functions, this system is believed to simply draw time values from the system hardware clocks within the nodes (computers) of the cluster for time reference and rely on the physical security of the system to prevent tampering.
Other systems provide a specified retention date within a data set that is locked against deletion or modification within a WORM storage implementation. This retention date scheme does not utilize any proprietary application program interfaces (APIs) or protocols, but rather, employs native functionality within conventional file (or other data containers, data sets or block-based logical unit numbers) properties available in commonly used operating systems. In an illustrative embodiment, the retention date/time is calculated by querying the file's last-modified time prior to commit, adding the retention period to this value and thereby deriving a retention date after which the file can be released from WORM. Prior to commit, the computed retention date is stored in the file's “last access time” property/attribute field, or another metadata field (typically open protocol-based) that remains permanently associated with the file and that, in being used for retention date, does not interfere with file management in a WORM state. Since this field is non-essential and rarely utilized in a WORM context, it can be adapted to store this date. Once stored, the retention date in this field is locked against modification. Where extension (never reduction) of a retention period is desired, the last access time field be updated, wherein the retention period extension is added to the existing last access time value to derive a new, later retention date for the file. Upon expiry of the retention date, the system allows deletion of the expired WORM file/data set.
The procedure checks whether the retention date provided by the administrator, user or interface is allowed by a particular set of WORM rules governing a volume. To this end, there may be more than one set of WORM rules applicable to a storage system. For storage requiring strict (government regulatory, for example) compliance, a strict or regulatory WORM volume may be employed. Storage in this volume is subjected to strict minimum default retention periods or, in the absence of a specified period, infinite periods. No modification or deletion of files is permitted until an applicable retention date expires. Conversely an “enterprise” volume, defined under a trusted administrator model may permit flexible setting of retention dates and/or no date as a default. Likewise, enterprise volumes may enable restoration from an earlier image of the file/volume, or destruction of a volume where appropriate.
In an illustrative embodiment, a secure compliance clock is used to gauge the time relative to the retention period. In one embodiment, this clock uses a trusted timebase, such as a repetitive file server process to derive its time value. The clock is tamper-proof, and its time value is guaranteed to be no sooner that actual time, but may be slightly longer to further ensure observance of the full retention period. In other embodiments, the clock can be based upon a trusted third-party clock (such as a government-based clock) or an additional secure hardware-based clock.
The stored retention date/time can overcome certain limitations in bit range for older open protocols by providing a base time date (epoch) that is relatively contemporary with the present day. The available date range under such bit range limitations can be extended generally by redefining the epoch (base time) to which the time value is relative. The identical format is used for dates within the epoch while wrap-around encoding of the date (e.g. using dates before the epoch as an extended range) for values appearing within the extended date range.
WORM Volume Creation
WORM volume creation is typically implemented by the administrator using the administrative console and interface to input the appropriate volume create command. The file system is modified to recognize a new semantic relative to the conventional (pre-WORM) volume create command that designates a WORM volume. The administrator initially creates the WORM volume having some general parameters such as for size in terms of number of disks. In one embodiment this is accomplished by typing in the following exemplary command including the volume name “volname” and the “-L”, the number of disks:
vol create <volname>-L<number of disks>.
Next, the volume is assigned the requested name (volname) on the file server. As such, it is identified as /vol/volname. The disks that are associated with the volume are marked with a WORM attribute on their RAID labels in persistent storage so that they are identified as WORM volumes. This may require a modification of the conventional RAID layer structure to account for this added label. While WORM is enforced primarily at the file system layer, this tag serves both to identify the assigned disk as a WORM store. Here, the WORM designation is stored in non-volatile memory in the persistent label of a WORM disk. The label contains identification information regarding the disk that permits the storage operating system to map the disk to particular volumes, and is stored in a standardized location on the disks (for example, the beginning) In this manner, the WORM designation is maintained on the disk itself containing the WORM file. Additionally, the WORM designation can be stored in NVRAM or other persistent/battery-backup memory structures and to prevent the inadvertent or deliberate destruction of the volume.
Deletion of Read Only Files
Accordingly, when a file is committed to WORM, it cannot be altered or deleted by applications, users, or administrators until the file retention date is reached. The exception is that some users may have the permission (e.g., administrative access), that allows that user only to a delete a file prior to it reaching its date scheduled for deletion (e.g., retention date). In some systems, this may be called a “privileged delete” feature.
The data that is committed to read only state generally cannot be changed or deleted before its retention date. However, one can change or delete the empty directories and files that are not committed to a WORM state on many file storage systems. Directories do not behave any differently than they would on regular volumes or files, with the exception that usually in file management systems, they cannot be renamed or moved once created. Generally, it may be a requirement for regulatory compliance that WORM data is not only non-erasable and non-rewritable, but it must also be locked down in the same location at which it was created. In the case of WORM implementation, this means that the directory path to WORM files must be locked down and should never change.
Generally, however, after the retention date is reached the WORM file is converted to a WORM expired file, and can be deleted by any users or by a software process. In some systems, the retention date on a WORM file is set when the file is committed to the WORM state, but it can be extended at any time. The retention period generally cannot be shortened for any WORM file.
Event Based Retention
Event based retention refers to retaining a record based on the occurrence of an event. This means that a record can only be deleted once an event occurs or a certain amount of time after an event occurs. For example, if the retention period of an insurance contract is set for Date of the Insured's Death plus 10 years, the event is triggered when the insured dies. For example, if the Insured Dies on 5 Mar. 2010, the contract must be retained until 4 Mar. 2020 and can only be deleted on or after 5 Mar. 2020.
Data storage management systems generally do not have a native ability to classify a file as an event based retention WORM file. Rather, a user must assign a retention period of infinity to a file prior to committing to the WORM state. Accordingly, those files are committed to a WORM state with a retention date of infinity, so they cannot be deleted at any time by a normal user. Then, once the event occurs and/or the relevant amount of time elapses, a privileged access delete feature or its equivalent must be used to delete the record at the appropriate time.
For example: If the retention period of the contract is set for Date of the Insured's Death plus 10 years and the insured dies on 5 Mar. 2010, the event is captured and examined by the application. The application infers that the contract needs to be retained for 10 years and flags the contract to be deleted, using privileged delete, on 5 Mar. 2020. If nothing occurs until 5 Mar. 2020, the application deletes the contract on 5 Mar. 2020. However if a legal hold is put on the contract on 15 Aug. 2016, the application captures this information and infers that the contract should not be deleted on 4 Mar. 2020 but rather it should be deleted when the legal hold is removed. The legal hold is removed on 11 Jan. 2021, and the application simply deletes the contract using the privileged delete feature.
In some WORM file systems, in order to set a file as a WORM file, normally a retention time (e.g., “set atime”) must be specified prior to setting the file as a WORM file. If it is not, the retention time of the new WORM file will be set to the default retention time of that volume. Accordingly, with this system, to set an event based retention file, the “set atime” must be set to infinity. Then, once the event occurs, the file must be flagged for deletion at a certain time. Finally, since the file's retention time is set to infinity, a privileged delete feature must be utilized to delete the volume when it is flagged for deletion.
In the drawings, the same reference numbers and any acronyms identify elements or acts with the same or similar structure or functionality for ease of understanding and convenience. To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the Figure number in which that element is first introduced.